Implement optimized catchup when wallet backend falls behind tip#428
Merged
aditya1702 merged 21 commits intoadd-backfillingfrom Dec 23, 2025
Merged
Implement optimized catchup when wallet backend falls behind tip#428aditya1702 merged 21 commits intoadd-backfillingfrom
aditya1702 merged 21 commits intoadd-backfillingfrom
Conversation
Contributor
|
Would it make sense for there to be periodic check points during live ingestion that can kick off optimized catch ups? |
| result.Duration = time.Since(start) | ||
| return result | ||
| } | ||
| batchBuffer.Clear() |
Contributor
There was a problem hiding this comment.
it could be nice to emit a metric for buffer size here so we can see how it relates to theBackfillDBInsertBatchSize
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
BackfillModeiota variable and do an optimized catchup when wallet backend has fallen behind network tip beyond a threshold.CatchupTestSuitethat tests stopping the wallet backend ingest container, letting the network go ahead and then doing optimized catchup using parallel workers.Why
This optimized catchup is helpful because for normal live ingestion, we process ledgers sequentially as they are produced. However in cases where wallet backend is behind the tip, we can intelligently fetch multiple ledgers from the ledger backend at once and process them parallelly (sort of like an intelligent backfill) and speed up catchup.
Known limitations
N/A
Issue that this PR addresses
#429